That sounds like you are mixing up (together) the functions of a part (component) and pcb. An svg file used for a pcb image contains only cutout shapes and (maybe silkscreen). It will not have anything related to connectors or copper.
A part (component) has the copper pads, silkscreen, and (optionally) drill holes. The holes can be for through hole connections (when they have a copper border (svg stroke width), or mounting holes when not stroke exists. A common recommendation is to mark mounting hole location of a part using only silkscreen. That way the part can be used without the mounting hole when desired, and the silkscreen shows where to place a separate “hole” component when it is wanted.
You need a board svg, and a separate part (or multiple parts).
Only tricky because the board does not do align to grid. Suggest putting a marker (silkscreen) on the board someplace that will be on the grid. Place a part (like a hole), align it to grid, then drag the board to match. Lock the board to keep it from moving, delete the part used to align the board, then place the real part.
That works for aligning a connector (that is in a part) to the grid of the current view, or connectors within a part relative to each other. It does not work for aligning board image with the grid. If the image is not aligned to the grid, the connector of the part will not be aligned with the image either.
You can place the board image using exact coordinates, but you need to know what (if any) the offset is from the image reference point (corner) to the connector position that is to be lined up with the grid. That offset will be consistent for a single board image, but could easily be different for 2 different boards. once the board image is positioned correctly (and locked) then parts can be positioned using the standard snap (align to grid), without needing to be placed by exact coordinates.